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ABSTRACT 



Mixed methodology was used in the formative evaluation of an 
integrated Arts- in-Education program during the first year of a 3 -year 
program involving four schools and three grade levels (grades 2, 6, and 10). 

A quantitative/qualitative survey instrument was administered to 
approximately 400 students to help assess the success of the program. Surveys 
were analyzed by coding all questions (quantitative and qualitative) into 
descriptive variables. New variables were formed during the analysis process, 
resulting in enriched outcomes and greater understandings. Strong student 
support and enthusiasm for the program, strong and positive evidence of 
integration of the arts curriculum content at all grade levels, and strong 
evidence of the curriculum's span of multiple intelligences and multiple 
learning areas were found. Implications for year two of the program and its 
evaluation are discussed. An appendix summarizes information about learning 
standards, multiple intelligences, and learning areas. (Contains 17 
references.) (Author/SLD) 
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quiuiumve, mio descriptive variables. New variables were formed during the 
analysis process, resulting in enriched outcomes and greater understandings. 
Implications for year two of the program and its evaluation are discussed. 




Introduction 



Programs that integrate the arts into core curricula are increasing in number 
as school budgets decrease, arts programs are cut, and/or the recognition of value in 
aesthetic education increases. Debate rages within the world of arts education 
concerning the merits of such integration; yet, little is known about the success or 
failure of programs that integrate arts into the core curricula. The lack of adequate 
numbers of research studies in this area is largely due to the unknown and unique 
nature of the variables under consideration. 

Qualitative methodologies involving extensive observations and immersion in 
the setting have been advocated as the most appropriate means of examining 
integration in a natural setting (Bresler, 1995). Some researchers are advocating the 
value of using mixed methodologies in these settings. A conceptual framework for 
such designs was developed by Greene, Caracelli, & Graham (1989) through their 
analysis of 57 empirical, mixed-method evaluations. Based upon their findings, this 
paper affirms the use of the multiple-method design for complementarity purposes. 
This method uses the qualitative and quantitative approach to measure overlapping 
but also different facets of a phenomenon, yielding an enriched, elaborated 
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understanding of that phenomenon. The purpose of this paper is to provide an 
increased understanding of the value in using mixed methodologies for evaluations of 
programs in the arts. 

Background 

While most educators acknowledge the value of arts education, there is much 
we do not know about the impact of teaching the arts. Elliot Eisner, advocate for the 
arts in education, points out that we do not know: 

whether there are certain critical periods in human development 

during which the ability to learn how to attend to and respond to the fine 
arts is optimal 

* whether neglect of the arts during these periods severely diminishes 
the probability that individuals will be able to respond to them at a later 
date 

* whether experience in arts activities fosters the development of 
imaginative abilities that transfer to other fields 

how effectively the arts can be taught in schools or classrooms where 
the pervasive atmosphere is anti-aesthetic and anti-imaginative 

what kind of people or society we could develop if schools spent as 
much time teaching the fine arts as they now spend teaching the three 
R’s 

* the extent to which teachers who themselves have limited 

backgrounds and abilities in the fine arts can teach them effectively 
(1980, pp. 598-599). y 

Psychologists and educators have worried about the answers to these and other 
questions relating to the study of art for many years. Lev Vygotsky stated in his 
Psychology of Art (1971, p. 259), “Psychological investigation reveals that art is the 
supreme center of biological and social individual process in society, that it is a 
method for finding an equilibrium between man and his world, in the most critical and 
important stages of his life”. John Dewey, in his Art os Experience (1934), espoused 
that aesthetic experience is the ultimate human experience and that its presence in 
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schools is one of the highest virtues of meaningful education. Eisner has more 
concretely outlined what children learn when they paint: they can create images 
from materials and feel satisfaction from so doing; the images they create can also be 
symbols, symbolic images can be used as vehicles for symbolic play; the process of 
image-making requires the making of judgments; images can be related to other 
images to form a whole; they can develop skills that make it possible for them to 
create illusion and to form images that are visually persuasive; ideas and emotions 
that are not physically present can be symbolized by images; there are ideas, images 
and feelings that can only be expressed through visual form; and, the world itself can 
be regarded as a source of aesthetic experience and as a pool of expressive form 
(1979a, pp. 109-115). 

Regarding the evaluation of arts programs and curricula in schools, Eisner has 
further commented: 

There is, of course, nothing wrong with knowing how well or how poorly 
a student performs. Yet schools, insofar as they are educational 
institutions, should not be content with performance. Education as a 
process is concerned with the cultivation of intellectual power, and the 
ability to determine what a student knows is not necessarily useful or 
sufficient for making that process more effective.. .they focus almost 
exclusively on the products of the enterprise.. .while they neglect the 
conditions, context, and interactions that led to these consequences In 
practical terms, they provide very little that is of use to the teacher in 
order to know what to alter or what to maintain in the course of teaching 
or in the design of the curriculum (1979b, pp. 11-12). 

Eisner feels that in our rush to “develop a science of education,” we have limited our 
conception of what constitutes cognition (1979b, p. 14). Essentially, what cannot be 
measured by numbers is considered noncognitive. The response to these issues has 
been the development of qualitative research methods, as advocated by Eisner 
through the concepts of educational connoissuership (the art of appreciation) and 
educational criticism (the artistic description of events, their interpretation and 
appraisal). 
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As the value of qualitative research has become more widely recognized 
(Denzin, 1978; Marshall & Rossman, 1989; Patton, 1990), the measurement of 
attitudes, feelings, and emotions of those involved in evaluations is also becoming 
more valued (Dereshiwsky & Packard, 1992). While a large body of research 
regarding Arts-in-Education (AIE) programs has not yet developed, the studies that 
do exist represent both the purely qualitative and the mixed-methodologies. 

A formative evaluation of a three-year AIE program in 11 urban schools used 
quantitative data in the form of self-report questionnaires, structured interviews, and 
structured classroom observations during the first year to compare student 
perceptions of the classroom learning environment, degree of program 
implementation, arts-related activities, and course evaluation. No program effects 
were discerned. Since evaluators felt there was a program impact, the second and 
third year evaluations were changed to semistructured, open-ended observation and 
interview instruments. As a result of the change in evaluative techniques, patterns 
became discernible from transformed qualitative data, and new variables were 
created through a merging of qualitative and quantitative data. It was discovered 
that the variable, “principal support,” was significantly correlated with the extent of 
program implementation (r= .74, pc.01)-a fact not apparent before the merging of 
both types of data (Talmage & Rasher, 1981). 

The Minnesota Center for Arts Education examined its interdisciplinary 
education efforts through a focus group research project in the spring of 1991. 
Twenty-six Arts High School students and fourteen teachers took part in meetings to 
identify factors that positively affect interdisciplinary education. Their findings 
included: flexible attitudes on the part of the teachers; adequate prepl anning time for 
the teachers; preservice and inservice education for teachers; use of a variety of 
participatory, hands-on teaching and learning strategies; discovery of natural 
linkages between areas; opportunities for team teaching; and, opportunities for 
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student cooperative work as well as individual work (Minnesota, 1992). 

A survey of 22 Arts in Education programs (Simmons, 1996) revealed a shift 
from short term projects to long term projects attributed to “the desire to realize 
more fully the potential of arts to transform students’ lives and education itself.” In 
addition, it was found that: participating teachers consistently developed greater 
confidence regarding artworks and a broader perspective on themselves, then- 
students, art, and pedagogy; and, greater emphasis needed to be placed on the role of 
the teaching artists’ development of pedagogy and the philosophy of aesthetic 
education. Similarly, the current evaluative study is a long-term project that 
addresses Eisner’s issue of utility for teachers. 

Methodology 

The ATE program evaluated in this study began during the fall of 1995 in four 
communities near Albany, New York. The purpose of this formative evaluation was 
to provide information for the program staff to use in the development of activities for 
the second and third years of the AIE program, and to provide initial assessment of 
curriculum integration, student outcomes, team building, and future needs for 
continued integration in the second program year. 

A mixed methods approach to evaluation was used to collect and integrate 
evaluation data. Measures taken to collect the data included: observations of artist 
residencies, staff planning sessions, and classrooms; administrator (arts coordinators, 
principals and superintendents), teacher, and artist interviews; and, paper/pencil 
surveys of second grade parents and of all involved students. Only the paper/pencil 
student survey and the classroom observations (relating to the objectives of 
curriculum integration, student outcomes, and future integration needs) are examined 
in this paper. The student survey, which obtained information from the student 
viewpoint, was distributed to the approximately 400 students involved in the project 
at four schools and at three grade levels (second, sixth, and tenth). Both the survey 
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and the classroom observations occurred during the spring of 1996. 

A data analysis strategy was developed which allowed descriptive analysis of 
all coded survey questions on SPSS (the Statistical Package for the Social Sciences). 
Initially, codings were based largely upon variables taken from the New York State 
Framework for the Arts (see Appendix). Where choices could be made between a 
more quantitative analysis versus a more qualitative analysis, the qualitative 
methodology (with its apparent greater applicability to the arts) was selected. 

Examination of the preliminary results from the student surveys revealed that 
they were not measuring the objectives adequately to provide meaningful feedback 
for future curriculum impact. New variables were then created for analyzing the 
classroom observations in order to broaden the contextual analysis of the qualitative 
data into explanations that teachers could more readily utilize (as suggested by 
Eisner, 1979b). The development of these new variables centered around key 
theories of intelligence and curriculum: Howard Gardner’s Theory of Multiple 
Intelligences and the Nowakowski, et al., model of learning areas (see Appendix). 
Results of all data analyses were interpreted and reported in an integrated fashion, 
resembling the Data Consolidation/Merging strategy developed by Caracelli & Greene 
(1993). 

Results & Implications 

“From a psychological point of view, the role of criticism is to organize the 
effects of art” (Vygotsky, 1971). Applying this statement to educational evaluation, 
Elliot Eisner might call this criticism “educational criticism” (1979b) and its effects 
would be organized through the results obtained from studies such as this AIE 
program evaluation. 

Major outcomes resulting from this study (related to the student survey and 
classroom observations) were determined to be: Strong student support and 
enthusiasm for the program; strong and positive evidence of integration of the arts 




6 



curriculum content at all grade levels; strong support of the New York State 
Framework for the Arts (and learning standards); and, strong evidences of the 
curriculum spanning the multiple intelligence and multiple learning areas across all 
three grade levels. In addition to the continued and enhanced support of all the major 
outcomes, needed improvements for year two of the AIE program curricul um became 
clear: the inclusion of more curriculum areas into the arts integration program; 
stronger emphasis on integration into the standard curriculum (reducing the need for 
pullout situations); recognition of the arts as cognitive (as well as affective) learning 
areas; and, recognition of the arts as consisting of both active and passive modes of 
learning. 

Implications for year two of the AIE program evaluation were: further 
refinement of the measures established, including expanded codings and expanded 
analysis of current variables; the addition of another set of grades (3rd, 7th, & 11th) 
the collection of longitudinal data; the addition of new methods of data collection 
(student interviews, teacher/artist focus groups) for triangulation purposes; raising 
the analytical methodology to a multidimensional level across instruments and 
constructs; and, continuing the search for new variables to further enrich the 
evaluative design for programs in the arts. 

From this first and most formative year, a foundation was established upon 
which could be built a complete structure-capable of recognizing the cognitive value 
in the arts and AIE programs, capable of providing the feedback and motivation 
needed by teachers and administrators for program continuance and improvement— 
and, capable of enriching future evaluations through the knowledge that new 
variables and methods reflecting higher levels of understanding and synthesis could 
continue to be developed for programs in arts education. 
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Appendix 



New York State Education Department (1994). Curriculum, instruction, and 
assessment preliminary draft framework for the arts. Albany, NY: Author. 

The four learning standards: 

1. Creating, performing, and participating in the arts; 

2. Knowing and using arts materials and resources; 

3. Responding to and analyzing works of art; 

4. Understanding the cultural dimensions and contributions of the arts. 



Gardner, H. (1983). Frames of mind: The theory of multiple intelligences. New 
York: Basic Books. 

The seven intelligences: 

Musical 

Bodily-Kinesthetic 

Logical-Mathematical 

Linguistic 

Spatial 

Interpersonal 

Intrapersonal 



Nowakowski, J., Bunda, M. A, Working, R., Bemacki, G. & Harrington, P. 
(1985). A handbook of educational variables: A guide to evaluation. Boston, MA: 
NijhofFPubhshing. 

The seven learning areas: 

Intellectual 

Emotional 

Physical & Recreational 

Aesthetic & Cultural 

Moral 

Vocational 

Social 
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